Extended reality systems for visualizing and controlling operating room equipment

ABSTRACT

A camera tracking system receives patient reference tracking information indicating pose of a patient reference array tracked by a patient tracking camera relative to a patient reference frame. A local XR headset view pose transform is determined between a local XR headset reference frame and the patient reference frame. Remote reference tracking information is received indicating pose of a remote reference array tracked by a remote reference tracking camera. A remote XR headset view pose transform is determined between a remote XR headset reference frame of a remote XR headset and the remote reference array. A 3D computer image is transformed from a local pose determined using the local XR headset view pose transform to a remote pose determined using the remote XR headset view pose transform. The transformed 3D computer image is provided to the remote XR headset for display with the remote pose relative to the remote XR headset reference frame.

FIELD

The present disclosure relates to surgical operating room equipmentoperations and computer assisted navigation of equipment and operatorsduring surgery.

BACKGROUND

Surgical operating rooms can contain a diverse range of medicalequipment, which can include computer assisted surgical navigationsystems, surgical robot systems, medical imaging devices (e.g.,computerized tomography (CT) scanners, magnetic resonance imagingscanners, fluoroscopy imaging, etc.), neuromonitoring equipment, patientmonitors, microscopes, anesthesia equipment, etc.

A computer assisted surgical navigation system can provide a surgeonwith computerized visualization of the present pose of a surgical toolrelative to medical images of a patient's anatomy. Camera trackingsystems for computer assisted surgical navigation typically use a set ofcameras to track a tool reference array on a surgical tool which isbeing positioned by a surgeon during surgery relative to a patientreference array attached to a patient. The reference array, alsoreferred to as a dynamic reference array (DRA) or dynamic reference base(DRB), allows the camera tracking system to determine a pose of thesurgical tool relative to anatomical structure within a medical imageand relative to the patient. The surgeon can thereby use real-timevisual feedback of the determined pose(s) to navigate the surgical toolduring a surgical procedure on the patient.

A surgical robot system can utilize optical tracking registered to amedical image as feedback for positioning a robotic arm while alsovisualizing instruments. The robotic arm includes an end effector whichmay be configured to guide a surgical tool used by a surgeon to performa surgical procedure on a patient. Additionally, many surgical workflowswith computer assisted surgical navigation systems and surgical roboticsystems require x-rays or computerized tomography (CT) scans duringoperation and/or registration procedures.

In view of the number and diversity of medical equipment, attempting toposition and control the medical equipment using numerous different userinterfaces before and during a surgical procedure can become overlycomplex especially while attempting to maintain sterility by minimizingtouching of surfaces of the medical equipment. Moreover, the medicalequipment is usually controlled through physical interfaces whichnecessitate that operators be proximately located thereto, and themedical equipment displays are usually configured for contextualobservation by operators proximately located thereto.

SUMMARY

Some embodiments of the present disclosure are directed to cameratracking systems and associated methods and computer program productsthat enable a remote operator who is wearing a remote extended reality(XR) headset to visualize and interact with three-dimensional (3D)computer images which are also viewable by another operator (localoperator) who is wearing a local XR headset while performing a surgicalprocedure on a patient. Moreover, the remote operator wearing the remoteXR headset may be able to visualize and control medical equipment thatis remote from the remote operator during use of the medical equipmentby the local operator.

In accordance with some embodiments, a camera tracking system thatincludes at least one processor (also referred to as “processor”) isoperative to receive patient reference tracking information indicatingpose of a patient reference array tracked by a patient tracking camerarelative to a patient reference frame. The processor determines a localXR headset view pose transform between a local XR headset referenceframe of a local XR headset and the patient reference frame using thepatient reference tracking information. The processor receives remotereference tracking information indicating pose of a remote referencearray tracked by a remote reference tracking camera, and determines aremote XR headset view pose transform between a remote XR headsetreference frame of a remote XR headset and the remote reference arrayusing the remote reference tracking information. The processortransforms a 3D computer image from a local pose determined using thelocal XR headset view pose transform to a remote pose determined usingthe remote XR headset view pose transform which outputs a transformed 3Dcomputer image, and provides the transformed 3D computer image to theremote XR headset for display with the remote pose relative to theremote XR headset reference frame.

Some other embodiments are directed to camera tracking systems andassociated methods and computer program products that enable XR headsetsto be used to visualize and control various types of medical equipments.

In accordance with some embodiments, a camera tracking system includesat least one processor (“processor”) operative to receive equipmentreference tracking information indicating poses of medical equipmentsand a patient reference array tracked by a tracking camera relative to areference frame. The processor determines an XR headset view posetransform between an XR headset reference frame of an XR headset and thereference frame using the equipment reference tracking information. Theprocessor obtains operator-gesture tracking information from thetracking camera indicating movement of an object relative to the XRheadset reference frame by an operator wearing the XR headset. Theprocessor selects an operational command from among a set of operationalcommands based on the operator-gesture tracking information, andprovides instructions to one of the medical equipments based on theoperational command that is selected.

Other camera tracking systems and corresponding methods and computerprogram products according to embodiments of the inventive subjectmatter will be or become apparent to one with skill in the art uponreview of the following drawings and detailed description. It isintended that all such additional camera tracking systems, methods. andcomputer program products be included within this description, be withinthe scope of the present inventive subject matter, and be protected bythe accompanying claims. Moreover, it is intended that all embodimentsdisclosed herein can be implemented separately or combined in any wayand/or combination.

DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example andare not limited by the accompanying drawings. In the drawings:

FIG. 1 is an overhead view of a potential arrangement for locations of asurgical robotic system including a surgical robot and tracking camera,and personnel wearing extended reality (XR) headsets during a surgicalprocedure, in accordance with some embodiments;

FIG. 2 illustrates the robotic system with the surgical robot and thecamera positioned relative to the patient according to some embodiments;

FIG. 3 illustrates a surgical robotic system in accordance with anexample embodiment;

FIG. 4 illustrates electronic components of a surgical robot inaccordance with some embodiments;

FIG. 5 illustrates a block diagram of electronic components of asurgical robot in accordance with some embodiments;

FIGS. 6A-6C respectively illustrate a surgical robot with anend-effector, an expanded view of the end-effector, and a surgical toolin accordance with some embodiments;

FIGS. 7A-7B respectively illustrate a C-arm image device and an O-armimaging device in accordance with some embodiments;

FIG. 8 illustrates an overhead view of a local arrangement of equipmentand remote arrangement of equipment enabling a remote operator who iswearing a remote XR headset to visualize and interact with 3D computerimages which are also viewable by a local operator who is wearing alocal XR headset while performing a surgical procedure on a patient, inaccordance with some embodiments;

FIG. 9 is a flowchart of operations by a camera tracking system forenabling a remote operator wearing a remote XR headset to visualize andinteract with 3D computer images which are also viewable by a localoperator wearing a local XR headset while performing a surgicalprocedure on a patient, in accordance with some embodiments;

FIG. 10 is a flowchart of operations by a camera tracking system fortransforming operator gestures tracked relative to the remoteenvironment to corresponding gesture paths relative to the localenvironment, accordance with some embodiments;

FIG. 11 is a flowchart of operations by a camera tracking system forcontrolling movement of an end effector of a surgical robot responsiveto a hand/stylus gesture by a remote operator, in accordance with someembodiments;

FIG. 12 is a flowchart of operations by a camera tracking system fortransforming graphical and/or textual information which has been enteredby a remote operator located in a remote environment to a pose which isdisplayed to a local operator wearing a local XR headset in a localenvironment, in accordance with some embodiments;

FIG. 13 illustrates an operator controlling movement of an end effectorof a surgical robot and/or controlling movement of the surgical robotbase using hand gestures which are tracked by a camera tracking systemin accordance with some embodiments;

FIG. 14 is a flowchart of operations by a camera tracking system forcontrolling movement of the end effector and/or the surgical robot baseof FIG. 13 using tracked hand gestures, in accordance with someembodiments; and

FIG. 15 illustrates a block diagram of a surgical system that includesan XR headset, a computer platform, a camera tracking system component,imaging devices, and a surgical robot which are operative in accordancewith some embodiments.

DETAILED DESCRIPTION

It is to be understood that the present disclosure is not limited in itsapplication to the details of construction and the arrangement ofcomponents set forth in the description herein or illustrated in thedrawings. The teachings of the present disclosure may be used andpracticed in other embodiments and practiced or carried out in variousways. Also, it is to be understood that the phraseology and terminologyused herein is for the purpose of description and should not be regardedas limiting. The use of “including,” “comprising,” or “having” andvariations thereof herein is meant to encompass the items listedthereafter and equivalents thereof as well as additional items. Unlessspecified or limited otherwise, the terms “mounted,” “connected,”“supported,” and “coupled” and variations thereof are used broadly andencompass both direct and indirect mountings, connections, supports, andcouplings. Further, “connected” and “coupled” are not restricted tophysical or mechanical connections or couplings.

The following discussion is presented to enable a person skilled in theart to make and use embodiments of the present disclosure. Variousmodifications to the illustrated embodiments will be readily apparent tothose skilled in the art, and the principles herein can be applied toother embodiments and applications without departing from embodiments ofthe present disclosure. Thus, the embodiments are not intended to belimited to embodiments shown, but are to be accorded the widest scopeconsistent with the principles and features disclosed herein. Thefollowing detailed description is to be read with reference to thefigures, in which like elements in different figures have like referencenumerals. The figures, which are not necessarily to scale, depictselected embodiments and are not intended to limit the scope of theembodiments. Skilled artisans will recognize the examples providedherein have many useful alternatives and fall within the scope of theembodiments.

Turning now to the drawing, FIGS. 1 and 2 illustrate a surgical robotsystem 100 in accordance with some embodiments. Surgical robot system100 may include, for example, a surgical robot 102, one or more robotarms 104, a display 110, an end-effector 112, for example, including aguide tube 114, and an end effector reference array which can includeone or more tracking markers. The surgical robot system 100 may includea patient reference array 116 with a plurality of tracking markers,which is adapted to be secured directly to the patient 210 (e.g., to abone of the patient 210). Another reference array 170 is attached orformed on an instrument. The surgical robot system 100 may also utilizea tracking camera 200, for example, positioned on a camera trackingsystem component 202. The camera tracking system component 202 can haveany suitable configuration to move, orient, and support the trackingcamera 200 in a desired position, and may contain a computer operable totrack pose of reference arrays. The tracking camera 200 may include anysuitable camera or cameras, such as one or more infrared cameras (e.g.,bifocal or stereophotogrammetric cameras), able to identify, forexample, active and passive tracking markers for various referencearrays attached as the patient 210 (patient reference array), endeffector 112 (end effector reference array), extended reality (XR)headset(s) 150 a-150 b worn by a surgeon 120 and/or a surgical assistant126, etc. in a given measurement volume viewable from the perspective ofthe tracking camera 200. The tracking camera 200 may track markers 170attached to an surgical tool or other instrument manipulated by a user.The tracking camera 200 may scan the given measurement volume and detectthe light that is emitted or reflected from the reference arrays inorder to identify and determine poses of the reference arrays inthree-dimensions. For example, active reference arrays may includeinfrared-emitting markers that are activated by an electrical signal(e.g., infrared light emitting diodes (LEDs)), and passive referencearrays may include retro-reflective markers that reflect infrared light(e.g., they reflect incoming IR radiation into the direction of theincoming light), for example, emitted by illuminators on the trackingcamera 200 or other suitable device.

As will be explained in further detail below, in some embodiments thecamera tracking system component 202 can operate to enable a remoteoperator who is wearing a remote XR headset to visualize and interactwith 3D computer images which are also viewable by a local operator whois wearing a local XR headset, e.g., HMD1 150 a and/or HMD2 150 b, whileperforming a surgical procedure on the patient. In some furtherembodiments, the remote operator wearing the remote XR headset may beable to visualize and control medical equipment that is remote from theremote operator during use of the medical equipment by the localoperator. In some additional or alternative embodiments, the cameratracking system component 202 can operate to enable the enable XRheadset, e.g., HMD1 150 a and/or HMD2 150 b, to be used to visualize andcontrol various types of medical equipments. The camera tracking systemcomponent 202 may be part of the surgical robot 102 or another systemcomponent.

The XR headsets 150 a and 150 b may each include tracking cameras thatcan track poses of reference arrays within their camera field-of-views(FOVs) 152 and 154, respectively. Accordingly, as illustrated in FIG. 1, the poses of reference arrays attached to various objects in the FOVs152 and 154 of the XR headsets 150 a and 150 b and a FOV 600 of thetracking cameras 200, e.g., mounted to an auxiliary tracking bar.

FIGS. 1 and 2 illustrate a potential configuration for the placement ofthe surgical robot system 100 in an operating room environment. Forexample, the robot 102 may be positioned near or next to patient 210.Although depicted near the head of the patient 210, it will beappreciated that the robot 102 can be positioned at any suitablelocation near the patient 210 depending on the area of the patient 210undergoing the operation. The tracking camera 200 may be separated fromthe robot system 100 and positioned at the foot of patient 210. Thislocation allows the tracking camera 200 to have a direct visual line ofsight to the surgical field 208. Again, it is contemplated that thetracking camera 200 may be located at any suitable position having lineof sight to the surgical field 208. In the configuration shown, thesurgeon 120 may be positioned across from the robot 102, but is stillable to manipulate the end-effector 112 and the display 110. A surgicalassistant 126 may be positioned across from the surgeon 120 again withaccess to both the end-effector 112 and the display 110. If desired, thelocations of the surgeon 120 and the assistant 126 may be reversed. Thetraditional areas for the anesthesiologist 122 and the nurse or scrubtech 124 remain unimpeded by the locations of the robot 102 and camera200. The anesthesiologist 122 can operate anesthesia equipment which caninclude a display 34.

With respect to the other components of the robot 102, the display 110can be attached to the surgical robot 102 and in other exampleembodiments, display 110 can be detached from surgical robot 102, eitherwithin a surgical room with the surgical robot 102, or in a remotelocation. End-effector 112 may be coupled to the robot arm 104 andcontrolled by at least one motor. In example embodiments, end-effector112 can comprise a guide tube 114, which is able to receive and orient asurgical instrument 608 (described further herein) used to performsurgery on the patient 210.

As used herein, the term “end-effector” is used interchangeably with theterms “end-effectuator” and “effectuator element.” The term “instrument”is used in a non-limiting manner and can be used interchangeably with“tool” to generally refer to any type of device that can be used duringa surgical procedure in accordance with embodiments disclosed herein.Example instruments include, without limitation, drills, screwdrivers,saws, dilators, retractors, probes, implant inserters, and implants suchas a screws, spacers, interbody fusion devices, plates, rods, etc.Although generally shown with a guide tube 114, it will be appreciatedthat the end-effector 112 may be replaced with any suitableinstrumentation suitable for use in surgery. In some embodiments,end-effector 112 can comprise any known structure for effecting themovement of the surgical instrument 608 in a desired manner.

The surgical robot 102 is operable to control the translation andorientation of the end-effector 112. The robot 102 is operable to moveend-effector 112 under computer control along x-, y-, and z-axes, forexample. The end-effector 112 can be configured for selective rotationabout one or more of the x-, y-, and z-axis, and a Z Frame axis (suchthat one or more of the Euler Angles (e.g., roll, pitch, and/or yaw)associated with end-effector 112 can be selectively computercontrolled). In some example embodiments, selective control of thetranslation and orientation of end-effector 112 can permit performanceof medical procedures with significantly improved accuracy compared toconventional robots that utilize, for example, a six degree of freedomrobot arm comprising only rotational axes. For example, the surgicalrobot system 100 may be used to operate on patient 210, and robot arm104 can be positioned above the body of patient 210, with end-effector112 selectively angled relative to the z-axis toward the body of patient210.

In some example embodiments, the pose of the surgical instrument can bedynamically updated so that surgical robot 102 can be aware of the poseof the surgical instrument at all times during the procedure.Consequently, in some example embodiments, surgical robot 102 can movethe surgical instrument to the desired pose quickly without any furtherassistance from a surgeon.

As used herein, the term “pose” refers to the position and/or therotational angle of one object (e.g., dynamic reference array,end-effector, surgical instrument, anatomical structure, etc.) relativeto another object and/or to a defined coordinate system. A pose maytherefore be defined based on only the multidimensional position of oneobject relative to another object and/or relative to a definedcoordinate system, based on only the multidimensional rotational anglesof the object relative to another object and/or to a defined coordinatesystem, or based on a combination of the multidimensional position andthe multidimensional rotational angles. The term “pose” therefore isused to refer to position, rotational angle, or combination thereof.

In some further embodiments, surgical robot 102 can be configured tocorrect the path of a surgical instrument guided by the robot arm 104 ifthe surgical instrument strays from the selected, preplanned trajectory.In some example embodiments, surgical robot 102 can be configured topermit stoppage, modification, and/or manual control of the movement ofend-effector 112 and/or the surgical instrument. Thus, in use, inexample embodiments, a surgeon or other user can operate the system 100,and has the option to stop, modify, or manually control the autonomousmovement of end-effector 112 and/or the surgical instrument.

Reference arrays can be formed on or connected to robot arm 104,end-effector 112, patient 210, and/or the surgical instrument to trackposes in 6 degree-of-freedom (e.g., position along 3 orthogonal axes androtation about the axes). In example embodiments, a reference arrayincluding a plurality of tracking markers can be provided thereon (e.g.,formed-on or connected-to) to an outer surface of the robot 102, such ason robot 102, on robot arm 104, and/or on the end-effector 112. Apatient reference array including one or more tracking markers canfurther be provided on the patient 210 (e.g., formed-on orconnected-to). An instrument reference array including one or moretracking markers can be provided on surgical instruments (e.g., ascrewdriver, dilator, implant inserter, or the like). The referencearrays enable each of the marked objects (e.g., the end-effector 112,the patient 210, and the surgical instruments 608) to be tracked by thetracking camera 200, and the tracked poses can be used to providenavigation guidance to a surgical procedure and/or used to controlmovement of the surgical robot 102 for guiding the end-effector 112and/or an instrument attached to the robot arm 104. In exampleembodiments, the surgical robot system 100 can use tracking informationcollected from each of the reference arrays to calculate the pose (e.g.,orientation and location), for example, of the end-effector 112, thesurgical instrument 608 (e.g., positioned in the tube 114 of theend-effector 112), and the relative position of the patient 210.

FIG. 3 illustrates further details of the surgical robot system 100 andtracking camera 200 of FIG. 1 . Referring to FIG. 3 the surgical robotsystem 100 includes the surgical robot 102 including a display 110,upper arm 306, lower arm 308, end-effector 112, vertical column 312,casters 314, tablet drawer 318, and ring 324 which uses lights toindicate statuses and other information. The tracking camera 200 issupported by the camera tracking system component 202.

FIG. 4 illustrates a base 400 which may be a portion of surgical robotsystem 100 and cabinet 106. Cabinet 106 may house certain components ofsurgical robot system 100 including but not limited to a battery 402, apower distribution module 404, a platform interface board module 406, acomputer 408, a handle 412, and a tablet drawer 414. The connections andrelationship between these components is described in greater detailwith respect to FIG. 5 .

FIG. 5 illustrates a block diagram of certain components of an exampleembodiment of surgical robot system 100. Surgical robot system 100 mayinclude platform subsystem 502, computer subsystem 504, motion controlsubsystem 506, and tracking subsystem 532. Platform subsystem 502 mayfurther include battery 402, power distribution module 404, platforminterface board module 406, and tablet charging station 534. Computersubsystem 504 may include computer 408, display 110, and speaker 536.Motion control subsystem 506 may include driver circuit 508, motors 510,512, 514, 516, 518, stabilizers 520, 522, 524, 526, end-effector 112,and controller 538. Tracking subsystem 532 may include position sensor540 and camera converter 542. System 100 may include a foot pedal 544that can be actuated to control movement of the end effector 112, e.g.,stop-start and/or regulate speed of movement, and tablet computer 546which provides a touch-display interface for operators of the surgicalrobot system 100.

Input power is supplied to surgical robot system 100 via a power supply548 which may be provided to power distribution module 404. Powerdistribution module 404 receives input power and is configured togenerate different power supply voltages that are provided to othermodules, components, and subsystems of surgical robot system 100. Powerdistribution module 404 may be configured to provide different voltagesupplies to platform interface board module 406, which may be providedto other components such as computer 408, display 110, speaker 536,driver circuit 508 to, for example, power motors 512, 514, 516, 518 andend-effector 112, motor 510, ring 324, camera converter 542, and othercomponents for surgical robot system 100 for example, fans for coolingthe electrical components within cabinet 106.

Power distribution module 404 may also provide power to other componentssuch as tablet charging station 534 that may be located within tabletdrawer 318. Tablet charging station 534 may be in wireless or wiredcommunication with tablet 546 for charging tablet 546. Tablet 546 may beused by a surgeon consistent with the present disclosure.

Power distribution module 404 may also be connected to battery 402,which serves as temporary power source in the event that powerdistribution module 404 does not receive power from power supply 548. Atother times, power distribution module 404 may serve to charge battery402 if necessary.

Other components of platform subsystem 502 may also include connectorpanel 320, control panel 322, and ring 324. Connector panel 320 mayserve to connect different devices and components to surgical robotsystem 100 and/or associated components and modules. Connector panel 320may contain one or more ports that receive lines or connections fromdifferent components. For example, connector panel 320 may have a groundterminal port that may ground surgical robot system 100 to otherequipment, a port to connect foot pedal 544 to surgical robot system100, a port to connect to tracking subsystem 532, which may compriseposition sensor 540, camera converter 542, and cameras 326 associatedwith camera tracking system component 202. Connector panel 320 may alsoinclude other ports to allow USB, Ethernet, HDMI communications to othercomponents, such as computer 408. In accordance with some embodiments,connector panel 320 may provide a wireless (e.g., WiFi 802.11, cellular4G, 5G, NR, etc.) and/or wired communication connection with extendedreality (XR) headsets 150 (e.g., 150 a and 150 b in FIG. 1 ) worn by thesurgeon 120, surgical assistant 126, anesthesiologist 122, and/or thenurse or scrub tech 124, etc.

Control panel 322 may provide various buttons or indicators that controloperation of surgical robot system 100 and/or provide informationregarding surgical robot system 100. For example, control panel 322 mayinclude buttons to power on or off surgical robot system 100, lift orlower vertical column 312, and lift or lower stabilizers 520-526 thatmay be designed to engage casters 314 to lock surgical robot system 100from physically moving. Other buttons may stop surgical robot system 100in the event of an emergency, which may remove all motor power and applymechanical brakes to stop all motion from occurring. Control panel 322may also have indicators notifying the user of certain system conditionssuch as a line power indicator or status of charge for battery 402.

Ring 324 may be a visual indicator to notify the user of surgical robotsystem 100 of different modes that surgical robot system 100 isoperating under and certain warnings to the user.

Computer subsystem 504 includes computer 408, display 110, and speaker536. Computer 504 includes an operating system and software to operatesurgical robot system 100. Computer 504 may receive and processinformation from other components (for example, tracking subsystem 532,platform subsystem 502, and/or motion control subsystem 506) in order todisplay information to the user. Further, computer subsystem 504 mayalso include speaker 536 to provide audio to the user.

Tracking subsystem 532 may include position sensor 540 and cameraconverter 542. Tracking subsystem 532 may correspond to camera trackingsystem component 202 including tracking camera 200 as described withrespect to FIG. 3 . Position sensor 540 may be tracking camera 200.Tracking subsystem may track the pose of certain markers that arelocated on the different components of surgical robot system 100 and/orinstruments used by a user during a surgical procedure. This trackingmay be conducted in a manner consistent with the present disclosureincluding the use of infrared technology that tracks the pose of activeor passive elements, such as LEDs or reflective markers, respectively.The pose of structures having these types of markers may be provided tocomputer 408 which may be shown to a user on display 110. For example, asurgical instrument having these types of markers and tracked in thismanner (which may be referred to as a navigational space) may be shownto a user in relation to a three dimensional image of a patient'sanatomical structure.

Motion control subsystem 506 may be configured to physically movevertical column 312, upper arm 306, lower arm 308, or rotateend-effector 112. The physical movement may be conducted through the useof one or more motors 510-518. For example, motor 510 may be configuredto vertically lift or lower vertical column 312. Motor 512 may beconfigured to laterally move upper arm 308 around a point of engagementwith vertical column 312 as shown in FIG. 3 . Motor 514 may beconfigured to laterally move lower arm 308 around a point of engagementwith upper arm 308 as shown in FIG. 3 . Motors 516 and 518 may beconfigured to move end-effector 112 in a manner such that one maycontrol the roll and one may control the tilt, thereby providingmultiple angles that end-effector 112 may be moved. These movements maybe achieved by controller 538 which may control these movements throughload cells disposed on end-effector 112 and activated by a user engagingthese load cells to move surgical robot system 100 in a desired manner.

Moreover, surgical robot system 100 may provide for automatic movementof vertical column 312, upper arm 306, and lower arm 308 through a userindicating on display 110 (which may be a touchscreen input device) thepose of a surgical instrument or component on three dimensional image ofthe patient's anatomy on display 110. The user may initiate thisautomatic movement by stepping on foot pedal 544 or some other inputmeans.

Turning now to FIGS. 6A-6C, the surgical robot system 100 relies onaccurate positioning of the end-effector 112, surgical instruments 608,and/or the patient 210 (e.g., patient reference array 116) relative tothe desired surgical area. In the embodiments shown in FIGS. FIGS.6A-6C, the reference arrays include tracking markers 118, 804 which arerigidly attached to a portion of the instrument 608 and/or end-effector112.

FIG. 6A depicts part of the surgical robot system 100 with the robot 102including base 106, robot arm 104, and end-effector 112. The otherelements, not illustrated, such as the display, marker tracking cameras,etc. may also be present as described herein. FIG. 6B depicts a close-upview of the end-effector 112 with guide tube 114 and a reference arraythat includes a plurality of tracking markers 118 rigidly affixed to theend-effector 112. In this embodiment, the plurality of tracking markers118 are attached to the end-effector 112 configured as a guide tube.FIG. 6C depicts an instrument 608 (in this case, a probe) with aplurality of tracking markers 804 rigidly affixed to the instrument 608.As described elsewhere herein, the instrument 608 could include anysuitable surgical instrument, such as, but not limited to, guide wire,cannula, a retractor, a drill, a reamer, a screwdriver, an insertioninstrument, a removal instrument, or the like.

In FIG. 6C, the reference array 612 functions as the handle 620 of theinstrument 608. Four markers 804 are attached to the handle 620 in amanner that is out of the way of the shaft 622 and tip 624.Stereophotogrammetric tracking by the tracking camera 200 of these fourmarkers 804 allows the instrument 608 to be tracked as a rigid body andfor the system 100 to precisely determine the location of the tip 624and the orientation of the shaft 622 while the instrument 608 is movedwithin view of tracking camera 200.

To enable automatic tracking of one or more instruments 608,end-effector 112, or other object to be tracked in 3D (e.g., multiplerigid bodies), the markers 118, 804 on each instrument 608, end-effector112, or the like, may be arranged asymmetrically with a knowninter-marker spacing. The reason for asymmetric alignment is so that itis unambiguous which marker 118, 804 corresponds to a particular pose onthe rigid body and whether markers 118, 804 are being viewed from thefront or back, i.e., mirrored. For example, if the markers 118, 804 werearranged in a square on the instrument 608 or end-effector 112, it wouldbe unclear to the system 100, 300, 600 which marker 118, 804corresponded to which corner of the square. For example, for theinstrument 608, it would be unclear which marker 804 was closest to theshaft 622. Thus, it would be unknown which way the shaft 622 wasextending from the array 612. Accordingly, each array 612 and thus eachinstrument 608, end-effector 112, or other object to be tracked shouldhave a unique marker pattern to allow it to be distinguished from otherinstruments 608 or other objects being tracked.

Asymmetry and unique marker patterns allow the tracking camera 200 andsystem 100 to detect individual markers 118, 804 then to check themarker spacing against a stored template to determine which instrument608, end-effector 112, or another object they represent. Detectedmarkers 118, 804 can then be sorted automatically and assigned to eachtracked object in the correct order. Without this information, rigidbody calculations could not then be performed to extract key geometricinformation, for example, such as instrument tip 624 and alignment ofthe shaft 622, unless the user manually specified which detected marker118, 804 corresponded to which position on each rigid body.

FIGS. 7A and 7B illustrate medical imaging systems 1304 that may be usedin conjunction with robot system 100 and/or navigation systems toacquire pre-operative, intra-operative, post-operative, and/or real-timeimage data of patient 210. Any appropriate subject matter may be imagedfor any appropriate procedure using the imaging system 1304. The imagingsystem 1304 may be any imaging device such as a C-arm 1308 device, anO-arm 1306 device, a fluoroscopy imaging device, a magnetic resonanceimaging scanner, etc. It may be desirable to take x-rays of patient 210from a number of different positions, without the need for frequentmanual repositioning of patient 210 which may be required in an x-raysystem. As illustrated in FIG. 7A, the imaging system 1304 may be in theform of a C-arm 1308 that includes an elongated C-shaped memberterminating in opposing distal ends 1312 of the “C” shape. C-shapedmember 1130 may further comprise an x-ray source 1314 and an imagereceptor 1316. The space within C-arm 1308 of the arm may provide roomfor the physician to attend to the patient substantially free ofinterference from x-ray support structure 1318. As illustrated in FIG.7B, the imaging system 1304 may include an O-arm imaging device 1306having a gantry housing 1324 attached to a support structure imagingdevice support structure 1328, such as a wheeled mobile cart 1330 withwheels 1332, which may enclose an image capturing portion, notillustrated. The image capturing portion may include an x-ray sourceand/or emission portion and an x-ray receiving and/or image receivingportion, which may be disposed about one hundred and eighty degrees fromeach other and mounted on a rotor (not illustrated) relative to a trackof the image capturing portion. The image capturing portion may beoperable to rotate three hundred and sixty degrees during imageacquisition. The image capturing portion may rotate around a centralpoint and/or axis, allowing image data of patient 210 to be acquiredfrom multiple directions or in multiple planes. Although certain imagingsystems 1304 are exemplified herein, it will be appreciated that anysuitable imaging system may be selected by one of ordinary skill in theart.

Using XR Headset for Remote Assistance and Controlling Medical Equipment

As was explained above, the numbers and diversity of medical equipmentwhich can be present in an operating room can make it complex toproperly position and control the equipment through numerous differentuser interfaces before and during a surgical procedure. Moreover, themedical equipment is usually controlled through physical user interfaceswhich necessitate that operators be proximately located thereto, andthere is a need to minimize or avoid unnecessary touching of physicaluser interfaces in order to maintain sterility.

Some embodiments of the present disclosure are directed to cameratracking systems and associated methods and computer program productsthat enable a remote operator who is wearing a remote XR headset tovisualize and interact with 3D computer images which are also viewableby another operator (local operator) who is wearing a local XR headsetwhile performing a surgical procedure on a patient. Moreover, the remoteoperator wearing the remote XR headset may be able to visualize andcontrol medical equipment that is remote from the remote operator duringsurgical use of the medical equipment by the local operator.

An XR headset may be configured to augment a real-world scene withcomputer generated XR images. The XR headset may be configured toprovide an augmented reality (AR) viewing environment by displaying thecomputer generated XR images on a see-through display screen that allowslight from the real-world scene to pass therethrough for combinedviewing by the user. Alternatively, the XR headset may be configured toprovide a virtual reality (VR) viewing environment by preventing orsubstantially preventing light from the real-world scene from beingdirectly viewed by the user while the user is viewing the computergenerated AR images on a display screen. An XR headset can be configuredto provide both AR and VR viewing environments. Thus, the term XRheadset can referred to as an AR headset and/or a VR headset.

Remote Assistance and Training Via XR and Machine Vision NavigatedSurgery

Navigated surgery introduces tracking information which is not presentin traditional surgeries, but the addition of timestamped andsynchronized sensor-rich XR headsets and visible light machine vision(MV) navigation systems enables users to be visually provided with ainformation-rich environment during pre-operative planning andinter-operative performance of a surgical procedure.

Various embodiments are explained that connect, share, interface andmanipulate information generated by medical equipment and/or useroperators in such a way that remote surgical assistance, training andprocedure reviews can be greatly improved.

These embodiments may enable visualizing, manipulating, sharing andprioritizing relevant information in such a way that user operators notphysically present in the operating room or testing lab can feel a senseof connectivity and immersion as if they were local during the surgicalprocedure. This increased connectivity and immersion greatly increasesthe effectiveness of user operators providing remote assistance tosurgeons, surgical assistants, etc. while also greatly improving thepotential of remote training applications.

Although various embodiments are described in the context of orthopedicsurgery, they are not limited to any type of surgery. Moreover, theembodiments are not limited to using visible light optical trackingsensors, but instead can be operate with tracking information providedby inertial sensors, etc.

Various embodiments are now described with reference to FIG. 8 .

FIG. 8 illustrates an overhead view of a local arrangement of equipmentin a local environment 800, such as an operating room, and remotearrangement of equipment in a temporally and/or spatially separatedremote environment 810 where a remote operator 812 who is wearing aremote XR headset 150 c is operationally able to visualize and interactwith 3D computer images which are also viewable by a local operator,e.g., surgeon 120, surgical assistant 126, etc. who is wearing a localXR headset 150 a, 150 b, etc. while performing a surgical procedure on apatient, in accordance with some embodiments. The remote XR headset 150c can include tracking cameras that operate to track poses of a remotereference array 814 and an instrument reference array 816. Alternativelyadditionally, another tracking camera can be proximately located to theremote operator 812 to tracking poses of tracking arrays which, forexample, are within the field-of-view of the remote operator 812. Theremote operator 812 may view information displayed on a physical displaydevice 818 which may include a microphone 820. In accordance with someembodiments, a camera tracking system is communicatively connected to atleast some of the local equipment in the local environment 800 (e.g.,OR) and at least some of the remote equipment in the remote environment810 through at least one network, e.g., private or public network suchas the Internet. The camera tracking system may be part of the cameratracking system component 202, the surgical robot 102, and/or anothercomponent residing in the local environment 800, the remote environment810, and/or another location connected to the network 820.

Moreover, the remote operator 812 wearing the remote XR headset 150 cmay be able to visualize and control medical equipment in the localenvironment 800 during use of the medical equipment by the localoperator 120, 126, etc.

As used herein, the term “remote” signifies an operator who is notphysically present in the operating room (OR) or testing lab in the (1)spatial sense (2) temporal sense or both (3) spatial and temporal sense.This means that communication can be real-time with a remote operator atanother location (1) or information can be recorded for playback andanalysis in the cases of (2) and (3). Generally speaking, remoteoperators of type (1) are more likely to be providing technicallyassistance or expert support while (2) and (3) are more likely to be foreither training purposes of for after-the-fact issue analysis (e.g.,technical problem reporting).

The camera tracking system may use tracking information and otherinformation from multiple XR headsets 150 a and 150 b such as inertialtracking information and optical tracking information as well as(optional) microphone information. The XR headsets 150 a and 150 boperate to display visual information and play-out audio information tothe wearer. This information can be from local sources (e.g., thesurgical robot 102, and other medical equipment in the local environment800), remote sources (e.g., patient medical image server), and/or otherelectronic equipment. The XR headsets 150 a and 150 b track apparatussuch as instruments, patient references and end effectors in 6degrees-of-freedom (6 DOF). They also track the hands of the wearer bytracking the position of, e.g., 24 recognizable points on the hands. TheXR headsets 150 a and 150 b may also operate to track hand poses andgestures to enable gesture based interactions with “virtual” buttons andinterfaces displayed through the XR headsets 150 a and 150 b and canalso interpret hand or finger pointing or gesturing as various definedcommands. Additionally, the XR headsets 150 a and 150 b may have a 1-10×magnification digital color camera sensor called a digital loupe.

As explained above, there can be, and often is, an “outside-in” machinevision navigation bar (tracking cameras 200) in the local environment800. The navigation bar tracks instruments and may include a colorcamera. The machine vision navigation bar generally has a more stableview of the environment because it does not move as often or as quicklyas the XR headsets 150 a and 150 b tend to move while positioned onwearers' heads. The patient reference array 116 is generally rigidlyattached to the patient with stable pitch and roll relative to gravity.This local rigid patient reference 116 can serve as a common referencefor reference frames relative to other tracked arrays, such as areference array on the end effector 112, instrument reference array 170,and reference arrays on the XR headsets 150 a and 150 b.

In some embodiments, one or more of the XR headsets 150 a and 150 b areminimalistic XR headsets that display local or remote information butinclude fewer sensors and are therefore more lightweight.

One or more 2D monitors (e.g., display 34 in FIG. 1 ) and computersystems may be provided in the local environment 800 for alternativetouch, mouse, and/or keyboard interfaces that are also viewable by localindividuals who are or are not wearing XR headsets. These 2D monitorsmay or may not be draped for sterility purposes.

In addition to live and recorded sensor information, there is alsoimportant local information in the form of the current software statelocated in, e.g., a navigation controller or cloud server. For example,a navigation “plan” for navigated implanting of screws and/or otherdevices may be viewed and adapted om navigation guidance informationthat is provided to the XR headsets 150 a and 150 b and/or 2D monitorfor display.

The machine vision cameras may generate color video streams of thepatient, cadaver, phantom, etc. The patient, cadaver, phantom, etc. mayalso be reconstructed by any combination of machine vision and colorcameras to generate a 3D surface model thereof.

In FIG. 8 , the medical equipment within the local environment 800 ispositioned locally adjacent to the patient and the human operators,e.g., surgeon, surgical assistance, trainee or support staff. Incontrast, the remote equipment within the remote environment 810 isremotely located from local environment OR 800 and is not directlyobservable by the tracking cameras of the camera tracking systemcomponent 202 and/or of the local XR headsets 150 a and 150 b. Theremote environment 810 may, for example be physically remote (e.g., theother side of the world) or temporally remote (e.g., in the samelocation but days later for training/visualization/understandingpurposes).

In one embodiment, a minimum equipment confirmation for a remoteenvironment 810 is a 2D monitor and user interface (e.g., touchscreen ormouse and keyboard) that enables a remote operator 812 to visualize andinteract with 3D computer images which are also viewable by a localoperator in the local environment 800 who is wearing a local XR headset150 a or 150 b while performing a surgical procedure on a patient. Forexample, the remote operator 812 may view one or more color or monocularvideo streams generated by cameras in the local environment 800 via thenetwork 830. In some embodiments, the remote operator 812 can also viewthe location of instruments relative to patient anatomy or CT scans.When the remote environment 810 is not temporally remote, the remoteoperator 812 can interact in real-time with the local operator(s), suchas by graphically highlighting, marking-up, and/or modifying informationthat is displayed to the local operators 120, 128, etc. via the local XRheadset 150 a or 150 b and/or a 2D display device.

For example, a surgical plan may be modified by the remote operator 812in response to something seen by the remote operator 812 on the liveviews or feedback received from surgical robot 102 end effector 112sensors or something seen in preoperative or intraoperative patientimage scans. A minimum remote environment can be extended via the use ofa microphone 820 and video camera sensors, e.g., in the remote XRheadset 150 c, which enhances communication by allowing local operatorsto hear and see guidance from the remote operator 812.

The potential of assistance and/or training applications can besignificantly enhanced by the operations enabling the remote operator toview and visually interact through the remote XR headset 150 c withinformation viewed by the local operator through the local XR headset150 a/150 b and, vice versa, for the local operator to view informationthe local XR headset 150 a/150 b generated by the remote operator. Theremote XR headset 150 c may operate to track pose of an instrument array816, e.g., stylus array, which may be manipulated by the remote operator812 to generate graphical information that is provided to the localoperator for viewing through the local XR headset 150 a/150 b.

With digital information being shared between the remote environment 810and the local environment 800 via the network 830, spatial informationcan be transformed, presented and manipulated in visually meaningful,intuitive and useful coordinate systems.

In some embodiments, the local XR headsets 150 a and 150 b and theremote XR headset 150 c are each tracked in “locally level” coordinatesystems using accelerometers in the respective headsets, which enablestracking to be performed relative to gravity. Gravity (pitch and roll)is presumed to be constant across the remote environment 810 and thelocal environment 800. Because of this, a single 4 DOF (X, Y, Z positionand heading) transformation can be applied for transformations relatingto the remote XR headset 150 c so that the displayed content isconfigured to float in front of the remote XR headset 150 c in roughlythe same location as the same content is displayed through the local XRheadsets 150 a and 150 b.

At least one processor can configured to receive tracking informationfrom tracking cameras which identifies poses of tracked reference arraysrelative to various defined reference frames, which may include thefollowing:

-   -   1) p-frame: local patient reference frame;    -   2) v-frame: the remote (or virtual) reference frame;    -   3) s-frame: the local XR headset 150 a/150 b reference frame        (e.g., for a surgeon or assistance wearing the local XR headset        150 a/150 b while performing a medical procedure on the        patient); and    -   4) a-frame: the remote XR headset 150 c reference frame (e.g.,        for a remote surgeon, assistance, or trainee wearing the remote        XR headset 150 c while viewing information relating to the        medical procedure on the patient).

The 6 DOF affine transformation between the s and p frames are estimatedby the local tracking apparatus as are the transformations between the aand v frames. The 6 DOF transformations are referred to as T^(p)s andand T^(v)a, respectively. In some embodiments, accelerometers in theremote XR headset 150 c enable the 3D content to be transformed from apose for displaying through the local XR headset 150 a/150 b to atransformed pose for displaying through the remote XR headset 150 c.Posing the 3D content in a desired location for remote user to viewthrough the remote XR headset 150 c may include posing the 3D content infront of the remote operator 812 (e.g., X and Y position) at anappropriate height (e.g., Z position) and with a desired heading or yaw(e.g., orientation about the up/down Z axis). Denoting the XYZtranslation vectors as r and the heading angles as φ, measurements arethen needed of rsp, ray, φsp, and φav.

The at least one processor (“processor”) computes a difference intranslation and heading between the local p-frame and the remotev-frame. When it is determined that a local XR headset 150 a/150 b iswithin a desired range of poses, the remote operator 812 may initiatecomputation, e.g., by pressing a button or performing a defined handgesture which is tracked by a tracking camera (e.g., part of the remoteXR headset 150 c) to compute these 4-DOF deltas as follows:

φ_(δ)=φ_(sp)−φ_(av)

r _(δ) =r _(sp) −r _(av)

The remote XR headset 150 c continues directly tracking T^(v)a directlyas independent ray and R_(av) translation and direction cosine matrixcomponents. The φ_(δ) and r_(δ) yaw and translation deltas are appliedto the virtual content in order to make it appear in roughly the samelocation for local and remote XR headset wearers. If the remote operator812 wants to see and interact with the content from a differentperspective than the local operator 120/126, the remote operator 812 caninitiate rotation of the content heading φ_(δ) via a defined remote ARheadset 150 c command to create an angularly offset, such as on theopposite side of a virtual bed, and interact with the local operator120/126 based on the angularly offset view.

With the operational ability to communicatively share all sensor,aligned tracking, and software state information between operators inthe local and remote environments 800 and 810, communication amongoperators becomes highly intuitive. 2D screen sharing can includegenerating virtual 2D screens which are viewed through XR headsets inone or both environments 800 and 810. Virtual 2D screens can begenerated for viewing the XR headsets to show the state of the otherenvironment's 2D monitor or general information. For example, remoteoperator 812 can view through remote XR headset 150 c a virtual 2Dscreen showing information generated for medical equipment within localenvironment 800 and vice versa. The shared information can include,without limitation, annotations and mark-ups of medical imagery,annotations or markups of still images captured from sensors, video chatfeeds or 2D renderings of remote AR content, etc.

The remote operator 812 wearing the remote XR headset 150 c can beoperationally provided multiple XR specific ways of interacting with thelocal operator 120/126. For example, a tool reference array 816, e.g.,on a stylus, can be used and shown as a “remote stylus” or “remote hand”via virtual content displayed on the local XR headset 150 a/150 b and/oron the 2D local display in the local environment 800. A virtualrepresentation of the remote user's head location can also be displayedon the local XR headset 150 a/150 b and/or on the 2D local display inthe local environment 800 for improved social interaction (e.g.,nodding, head shaking or exact perspective become intuitively apparent).With the ability to point out detailed information or draw 3D virtualmark-ups via stylus (e.g., where to make incisions or place aquatrospike), point or gesture with hands and head without temporal andphysical proximity restriction, a remote operator 812 (e.g., experts orclinical representatives) can be virtually present in the OR and offerlive support.

In some embodiments, live data feeds of digital information are sharedbetween the local and remote environments 800 and 810 during navigatedprocedures, which can enhance communication between on-site and remotestaff and allows for improved training and assistance during and aftersurgeries. Examples of digital information which can be shared include:

a. Navigation video feeds: automated diagnostic and information mark-upsdraw attention to important information, views show everything that thenavigation cameras can see and are in visible light.

b. Color navigation view and digital loupe: a live perspective of thenavigation camera as well as what the surgeon/physician's assistant islooking at in full color and high resolution.

c. Instrument and end effector tracking: enables the remote operator tosee in 3D where all instruments, end effectors and other surgicalapparatus are in real-time.

d. Remote “stylus” tracking: allows remote operator to point out objectswith an accurately tracked stylus pose.

e. Hand tracking: allows remote operator to point out objects using handgestures in an intuitive manner.

f. Head tracking: allow local and remote operators to visually observewhere each other is looking relative to the patient and medicalequipment.

g. Plan information, timers, notes checklists and metadata:synchronizing such information via the network 830 enable the remoteoperator 812 to know the state of the planned versus executed surgery atall times and update plans or details accordingly.

h. CT data and other medical imagery: important CT and other 2D and 3Dmedical imagery can be viewed and marked-up in real-time (data mayinclude planned implant placement) by the remote operator 812 via theremote XR headset 812.

i. 3d surface reconstructions: reconstructions of the scene in 3d viamachine vision cameras can add to the information and sense of immersionor perspective of the remote operator 812.

The remote operator 812 may directly remotely control medical equipmentin the local environment 800 and/or provide textual and/or graphicalrecommendations/instructions to the local operator(s) 120 and 126 viahand gestures and/or movement of the instrument (stylus) reference array816 tracked by the remote XR headset 150 c and/or another trackingcamera.

Various camera tracking systems are now described which transform thelocal XR headset 150 a/150 b view of a 3D computer image for displaythrough the remote XR headset 150 c relative to the remote referencearray 814.

FIG. 9 is a flowchart of operations by a camera tracking system forenabling a remote operator wearing a remote XR headset 150 c tovisualize and interact with 3D computer images which are also viewableby a local operator wearing a local XR headset 150 a/150 b whileperforming a surgical procedure on a patient, in accordance with someembodiments.

Referring to FIG. 9 , camera tracking system includes at least oneprocessor (“processor” for brevity) operative to receive 900 patientreference tracking information indicating pose of a patient referencearray 116 tracked by a patient tracking camera 200 relative to a patientreference frame 116. The processor is further operative to determine 902a local XR headset view pose transform between a local XR headsetreference frame of a local XR headset 150 a/150 b and the patientreference frame using the patient reference tracking information. Theprocessor is further operative to receive 904 remote reference trackinginformation indicating pose of a remote reference array 814 tracked by aremote reference tracking camera, e.g., part of remote XR headset 816).The processor is further operative to determine 906 a remote XR headsetview pose transform between a remote XR headset reference frame of aremote XR headset 150 c and the remote reference array using the remotereference tracking information. The processor is further operative totransform 908 a 3D computer image from a local pose determined using thelocal XR headset view pose transform to a remote pose determined usingthe remote XR headset view pose transform which outputs a transformed 3Dcomputer image. The processor is further operative to provide 910 thetransformed 3D computer image to the remote XR headset 150 c for displaywith the remote pose relative to the remote XR headset reference frame.

The processor may be further operative to transform the 3D computerimage from the local pose to the remote pose while the patient trackingcamera is remote from the remote reference tracking camera 150 c and notpositioned to track pose of the remote reference array 814, and whilethe remote reference tracking camera 150 c is not positioned to trackpose of the patient reference array 116, e.g., because the localenvironment 800 and the remote environment 810 are spatially and/ortemporarily offset.

As explained above, a 4 degree-of-freedom (DOF) transformation can beused instead of a 6 DOF transformation using an accelerometer matters inthe local and remote XR headsets 150 a/150 b and 150 c and an assumptionthat the local and remote environments 800 and 810 are subject to thesame gravity vector. Using a 4 DOF transformation can substantiallyreduce the computing and memory resources that would otherwise berequired for performing a 6 DOF transformation at a frequency thatallows real-time update of displayed information. Accordingly, in oneembodiment the processor is further operative to determine a 4 DOF poseof the remote XR headset based on measured movement along threeorthogonal axes of the remote XR headset reference frame and rotationabout one of the three orthogonal axes aligned with gravitationaldirection. The operation to transform the 3D computer image from thelocal pose determined using the local XR headset view pose transform tothe remote pose determined using the remote XR headset view posetransform, includes processing the 4 DOF pose of the remote XR headsetthrough the remote XR headset view pose transform.

Some further embodiments are directed to identifying a remote pose of apath gesture performed by a remote operator wearing the remote XRheadset 150 c relative to the remote XR headset reference frame,transforming the remote pose of the path gesture relative to the remoteXR headset reference frame to a local pose relative to the local XRheadset reference frame, and providing a computer generated indicationof the path gesture with the local pose to the local XR headset 150a/150 b for display relative to the patient reference array 116. FIG. 10is a flowchart of corresponding operations that can be performed by acamera tracking system in accordance with some embodiments. Referring toFIG. 10 the processor of the camera tracking system is further operativeto obtain 1000 remote operator-gesture tracking information from theremote reference tracking camera 150 c indicating movement of an object816 (e.g., tracked stylus, hand, etc.) relative to the remote XR headsetreference frame by a remote operator 812 wearing the remote XR headset150 c. The processor determines 1002 a remote gesture path relative tothe remote XR headset reference frame based on processing the remoteoperator-gesture tracking information through the remote XR headset viewpose transform, and transforms 1004 the remote gesture path to a localgesture path relative to the local XR headset reference frame using thelocal XR headset view pose transform.

In some further embodiments, the processor provides the local gesturepath to the local XR headset 150 a/150 b for display relative to thelocal XR headset reference frame.

In another embodiment, while the remote operator 812 is viewing thetransformed 3D computer image displayed by the remote XR headset 150 c,remote operator moves the object 816 to indicate a remote gesture pathfor viewing by the local operator 120/126. In one embodiment, theprocessor is further operative to determine 1002 the remote gesture pathrelative to the remote XR headset reference frame based on trackingmovement indicated by the remote operator-gesture tracking informationof a hand and/or a stylus which is moved by the remote operator 812while concurrently viewing the transformed 3D computer image through theremote XR headset 150 c relative to the hand and/or stylus being moved.

Some further embodiments, the remote operator 812 can move the handand/or stylus to form a gesture which is recognized by the cameratracking system is corresponding to various defined operationalcommands, which can control equipment in the local environment 800,e.g., local to the patient reference frame. In one embodiment, theprocessor is further operative to select an operational command fromamong a set of operational commands based on the remote gesture pathcorresponding to defined gesture associated with the operationalcommand, wherein the operational commands in the set are associated withdifferent shaped gesture paths. The processor then provides theoperational command to an equipment, e.g., surgical robot 102, which islocal to the local XR headset.

In a further embodiment, the processor selects the operational commandfor relocating an end effector 112 connected to a surgical robot arm 104that is movable under control of a surgical robot system 100, from amongthe set of operational commands based on the remote gesture pathcorresponding to the defined gesture associated with the operationalcommand for relocating the end effector 112. FIG. 11 is a flowchart ofoperations by camera tracking system for controlling movement of an endeffector 112 of a surgical robot 102 responsive to a hand/stylus gestureby a remote operator 812, in accordance with some embodiments. Referringto FIG. 11 , the processor determines 1100 a present pose of the endeffector 112 based on end effector tracking information indicating poseof the end effector 112 tracked by the patient tracking camera 200relative to the patient reference frame. The processor controls 1102movement of the end effector 112 by the surgical robot system 100 fromthe present pose to a target pose relative to the patient referenceframe based on the operational command for relocating the end effector112.

In a further embodiment, the processor determines a planned end effectortrajectory path from the present pose to the target pose based on atleast a segment of the remote gesture path. The processor controlsmovement of the end effector by the surgical robot system to conform tothe planned end effector trajectory path from the present pose to thetarget pose.

The transformed 3D computer image may include a graphical representationof the end effector displayed based on the remote pose relative to theremote XR headset reference frame and include a graphical representationof anatomical structure of the patient displayed based on the remotepose relative to the remote XR headset reference frame. The processormay then be operative to determine a planned end effector 112 trajectorypath from a present graphical pose of the graphical representation ofthe end effector 112 to a target graphical pose of the graphicalrepresentation of the end effector 112 based on tracking movement offingers and/or a hand of the remote operator 812 wearing the remote XRheadset 150 c relative to the graphical representation of the endeffector displayed relative to remote XR headset reference frame, andcontrol movement of the end effector 112 by the surgical robot system100 to conform to the planned end effector trajectory path from thepresent pose relative to the patient reference frame to the target poserelative to the patient reference frame.

In some further embodiments the processor is operative to select theoperational command from among the set of operational commands whichcontrol at least one of the following:

-   -   1) operational settings of a computer assisted surgical        navigation system;    -   2) operational settings of a surgical robot system;    -   3) operational settings of medical imaging equipment which is        operable to obtain medical images of anatomical structure of the        patient;    -   4) operational settings of intraoperative neuromonitoring        equipment which is operable to monitor neural structures of the        patient;    -   5) operational settings of the local XR headset;    -   6) operational settings of a computer display which is local to        the local XR headset;    -   7) operational settings of a microscope which is local to the        local XR headset;    -   8) operational settings of an exoscope which is local to the        local XR headset;    -   9) operational setting of a lighting apparatus which is operable        to illuminate the patient;    -   10) operational settings of a powered adjustable surgical bed        which is operable to    -   support the patient;    -   11) operational settings of a microscope which is local to the        local XR headset;    -   12) operational settings of anesthesia equipment which is        operable to supply anesthesia to the patient;    -   13) operational settings of a clock and/or timer which is local        to the local XR headset;    -   14) operational settings of communication equipment which is        local to the local XR headset; and    -   15) operational settings of sound equipment which is local to        the local XR headset.

FIG. 12 is a flowchart of operations by camera tracking system fortransforming graphical and/or textual information which has been enteredby a remote operator 812 located in the remote environment 810 to a posewhich is displayed to a local operator 120/126 wearing a local XRheadset 150 a/150 b in a local environment, in accordance with someembodiments. Referring to FIG. 12 , the processor is operative to obtain1200 graphical and/or textual information entered by the remote operator812 and which is displayed by the remote XR headset 150 c with a remoteinformation pose relative to the remote XR headset reference frame usingthe remote XR headset view pose transform. The processor transforms 1202the remote information pose of the graphical and/or textual informationto a local information pose using the local XR headset view posetransform. The processor provides the graphical and/or textualinformation to the local XR headset 150 a/150 c for display with thelocal information pose relative to the local XR headset reference frame.

Some further embodiments are directed to operations that correlate videoframes of what is being viewed through the local XR headset 150 a/150 band viewed through the remote XR headset 150 c to ensure that theassociated operators are viewing time synchronized information. In somefurther embodiments, the processor is operative to correlate in timeindividual ones of video frames of a local video stream received fromthe patient tracking camera 200 with individual ones of video frames ofa remote video stream received from the remote reference trackingcamera, e.g., part of remote XR headset 150 c. The processor controlstiming when the individual ones of the video frames of the local videostream are provided to the remote XR headset 150 c for display based onthe correlation, and controls timing when the individual ones of thevideo frames of the remote video stream are provided to the local XRheadset 150 a/150 b for display based on the correlation.

Operating Room Equipment Visualizations and Control Using XR Headset(s)

Some other embodiments are now described which are directed to cameratracking systems and associated methods and computer program productsthat enable XR headsets to be used to visualize and control varioustypes of medical equipments.

Positioning and sterility may require a touch free method forcontrolling medical equipment which may be within reach of an operatoror beyond reach. Some embodiments are directed to operations that enablean operator wearing an XR headset to perform hand gestures which areviewed through the XR headset relative to the equipment to becontrolled. The hand gestures are tracked by a tracking camera, whichmay be part of the XR headset, and are recognized by camera trackingsystem as a command for controlling the proximately located equipment.Information generated by the equipment, such as patient medicalmeasurements and/or operational data, can be displayed through the XRheadset with a pose that is anchored proximately located to theassociated equipment. In this manner, an operator wearing the XR headsetcan intuitively view information from various equipment within an OR andmay further control operations of the equipment.

With continued reference to FIG. 8 , the associated operations can beperformed by a tracking camera that provides tracking information to acamera tracking system which is operative to control equipment andprovide graphical and/or textual information for display through the XRheadset. As explained above, the tracking camera may be part of the XRheadset 150 a/150 b and/or may be part of an auxiliary camera trackingbar 200.

Movement of medical equipment by the camera tracking system may beperformed relative to the patient reference array 116, so as to enableoperator gesture based controlled movement of equipment to operatordesired poses of the medical equipment relative to the patient.

FIG. 13 illustrates an operator controlling movement of an end effectorof a surgical robot and/or controlling movement of the surgical robotbase using hand gestures which are tracked by a camera tracking systemin accordance with some embodiments.

Referring to FIG. 13 , in one embodiment the camera tracking systemenables an operator to use hand gestures to move an end effector of asurgical robot. The camera tracking system tracks and recognizes aninitial gesture by the operator who is pointing a hand-palm or fingerfrom node point 1301 and which the camera tracking system projects alongpath 1302 to intercept the end effector (e.g., 112 in FIG. 3 ) at astart end effector location node 1304. The camera tracking systemsfurther tracks and recognizes movement of the operator's fingers, e.g.,opening from a pinch gesture, to extend to node point 1303 which thecamera tracking system projects along path 1303 to define a target endeffector location node 1306. The camera tracking system may display agraphical indication of the planned trajectory via the XR headset alongwhich the end effector is planned to be moved from the starting endeffector location node 1304 to the target end effector location node1306. Responsive to the operator indicating acceptance of the plannedtrajectory, e.g., by pressing and holding-down a foot pedal or byforming another defined hand gesture, the camera tracking system cancontrol motors of the surgical robot (e.g., 100 in FIG. 3 ) to move theend effector from the starting end effector location node 1304 to thetarget end effector location node 1306 along the planned trajectory.

With continued reference to FIG. 13 , in another embodiment the cameratracking system enables an operator to use hand gestures to movelocation of a surgical robot base, e.g., to position the surgical robotwith the desired poses relative to a patient reference frame (e.g., 116in FIG. 8 ). The camera tracking system tracks and recognizes an initialgesture by the operator who is pointing a hand-palm or finger from nodepoint 1301 and which the camera tracking system projects along path 1310to intercept a base (e.g., 106 in FIG. 3 ) of the surgical robot (e.g.,100 in FIG. 3 ) at a start base location node 1314. The camera trackingsystems further tracks and recognizes movement of the operator'sfingers, e.g., opening from a pinch gesture, to extend to node point1303 which the camera tracking system projects along path 1312 to definea target base location node 1316. The camera tracking system may displaya graphical indication of the planned trajectory via the XR headsetalong which the robot base is planned to be moved from the starting baselocation node 1314 to the target base location node 1316. Responsive tothe operator indicating acceptance of the planned trajectory, e.g., bypressing and holding-down a foot pedal or by forming another definedhand gesture, the camera tracking system can control motors connected towheels of the robot base to move the robot base from the starting baselocation node 1314 to the target base location node 1316 along theplanned trajectory.

FIG. 14 is a flowchart of operations by a camera tracking system forcontrolling movement of the end effector (e.g., 112 in FIG. 8 ) and/orthe surgical robot base (e.g., 106 in FIG. 3 ) using tracked handgestures, in accordance with some embodiments.

Referring to FIG. 14 , the camera tracking system includes at least oneprocessor (“processor”) operative to receive 1400 equipment referencetracking information indicating poses of medical equipments and apatient reference array tracked by a tracking camera relative to areference frame. The processor is operative to determine 1402 an XRheadset view pose transform between an XR headset reference frame of anXR headset and the reference frame using the equipment referencetracking information. The processor is operative to obtain 1404operator-gesture tracking information from the tracking cameraindicating movement of an object relative to the XR headset referenceframe by an operator wearing the XR headset. The processor is operativeto select an operational command from among a set of operationalcommands based on the operator-gesture tracking information, and toprovide instructions to one of the medical equipments based on theoperational command that is selected.

As explained above, the tracking camera may be part of the XR headset,and the reference frame may thereby be the same as the XR headsetreference frame.

In a further embodiment, the processor is operative to determine agesture path relative to the XR headset reference frame based onprocessing the operator-gesture tracking information through the XRheadset view pose transform, and to select the operational command fromamong the set of operational commands based on identifying that thegesture path corresponds to a defined gesture associated with theoperational command, wherein the operational commands in the set areassociated with different shaped gesture paths.

In a further embodiment, the processor is operative to select theoperational command for relocating an end effector connected to asurgical robot arm that is movable under control of a surgical robotsystem, from among the set of operational commands based on the gesturepath corresponding to the defined gesture associated with theoperational command for relocating the end effector. The processor isoperative to determine a present pose of the end effector based on endeffector tracking information indicating pose of the end effectortracked by the tracking camera relative to the reference frame, and tocontrol movement of the end effector by the surgical robot system fromthe present pose to a target pose relative to the reference frame basedon the operational command for relocating the end effector.

In a further embodiment, the processor is operative to determine aplanned end effector trajectory path from the present pose to the targetpose based on at least a segment of the gesture path, and to controlmovement of the end effector by the surgical robot system to conform tothe planned end effector trajectory path from the present pose to thetarget pose.

The processor may be operative to determine the planned end effectortrajectory path based on tracking movement of fingers and/or a hand ofthe operator wearing the XR headset relative to the end effector.

In a further embodiment, the processor is operative to select theoperational command for relocating medical imaging equipment in a roomunder control of a computer system, from among the set of operationalcommands based on the gesture path corresponding to the defined gestureassociated with the operational command for relocating the medicalimaging equipment. The processor is operative to determine a presentlocation in the room of the medical imaging equipment relative to thereference frame based on the equipment reference tracking information,and to determine a target location in the room for the medical imagingequipment relative to the reference frame based on at least a segment ofthe gesture path. The processor is operative to control movement of themedical imaging equipment by the computer system from the presentlocation to the target location based on the operational command forrelocating the medical imaging equipment.

In a further embodiment, the processor is operative to select theoperational command from among the set of operational commands whichcontrol at least one of the following:

-   -   1) operational settings of a computer assisted surgical        navigation system;    -   2) operational settings of a surgical robot system;    -   3) operational settings of medical imaging equipment which is        operable to obtain medical images of anatomical structure of a        patient;    -   4) operational settings of intraoperative neuromonitoring        equipment which is operable to monitor neural structures of a        patient;    -   5) operational settings of the XR headset;    -   6) operational settings of a computer display;    -   7) operational settings of a microscope;    -   8) operational settings of an exoscope;    -   9) operational setting of a lighting apparatus which is operable        to illuminate a patient;    -   10) operational settings of a powered adjustable surgical bed        which is operable to support a patient;    -   11) operational settings of a microscope;    -   12) operational settings of anesthesia equipment which is        operable to supply anesthesia to a patient;    -   13) operational settings of a clock and/or timer;    -   14) operational settings of communication equipment; and    -   15) operational settings of sound equipment.

In a further embodiment, the processor is operative to obtain firstgraphical and/or textual information from a first one of the medicalequipments, and second graphical and/or textual information from asecond one of the medical equipments. The processor displays the firstgraphical and/or textual information through the XR headset with a posein the XR headset reference frame defined to be adjacent to the firstone of the medical equipments and display the second graphical and/ortextual information through the XR headset with another pose in the XRheadset reference frame defined to be adjacent to the second one of themedical equipments.

The equipment information may be displayed through the XR headset with apose that is anchored relative to the associated equipment. In thismanner, the operator may look toward a particular equipment to initiatedisplay of the related information with the defined pose relative to theparticular equipment. An operator may use one or more hand gestures tocontrol what types of equipment information is displayed, size of thedisplayed information, and where the displayed information is posedrelative to the equipment. An operator may use various defined types ofhand gestures to control corresponding settings of the equipment, suchas one or more operational threshold levels used by the equipment.

In some further embodiments, the camera tracking system may scan theroom to automatically identify medical equipment which is present withinthe field of view of the tracking cameras. The camera tracking systemmay process various video streams from one or more XR headsets 150 a/150b and/or mounted to an auxiliary tracking bar 200 to identify medicalequipment. For example, the camera tracking system may determine amedical equipment type, model number, and/or a unique identifiercaptured in camera video stream(s) based on identifying a tag or othermachine-readable code on the medical equipment and/or based onidentifying a tracking array on the medical equipment. Alternatively oradditionally, the camera tracking system may identify medical equipmentbased on matching the shape observed in the camera video stream(s) to adefined geometric shape template for the medical equipment.

The camera tracking system may identify a pose of the medical equipmentwithin the room, and may enable an operator to use a hand gesture toidentify a target location for where the medical equipment is to bemoved. The camera tracking system may then determine a plannedtrajectory path for moving the medical equipment from the present poseto the target pose, and may display the plan trajectory path through oneof the XR headsets 150 a/150 b for approval by an operator. The cameratracking system may then control movement of the medical equipment fromthe present pose to the target pose, such as to position the medicalequipment relative to a patient reference array. The human trackingsystem may also identify in the camera video stream(s) obstacles, suchas power lines and/or communication lines extending along the floor, atable, etc., in a path between the present pose and target pose of themedical equipment, and may determine the plan trajectory path to have ashape that avoids such obstacles.

In this manner, the camera tracking system can operate to trackworld-anchored content in an intuitive manner for viewing by surgeonsand other operators during a surgical procedure. A surgical assistantmay adjust a surgeon's XR headset parameters from the other side of thebed using hand gestures to interact with a virtual head stabilizedinterface, and/or may adjust tracking camera operational modes oroutputs using hand gestures to interact with a virtual interfacedisplayed adjacent to or overlapping the tracking camera.

FIG. 15 illustrates a block diagram of a surgical system that includesan XR headset 150, a computer platform 1500, imaging devices, and asurgical robot 102 which are configured to operate in accordance withvarious embodiments.

The imaging devices may include the C-arm imaging device 1304, the O-armimaging device 1306, and/or a patient image database 1530. The XRheadset 150 provides an improved human interface for performingnavigated surgical procedures. The XR headset 150 can be configured toprovide functionalities, e.g., via the computer platform 1500, thatinclude without limitation any one or more of: identification of handgesture based commands, display XR graphical objects on a display device1512. The display device 1512 may a video projector, flat panel display,etc. The user can view the XR graphical objects as an overlay anchoredto particular real-world objects viewed through a see-through displayscreen. The XR headset 150 may additionally or alternatively beconfigured to display on the display device 1512 video streams fromcameras mounted to one or more XR headsets 150 and other cameras.

Electrical components of the XR headset 150 can include a plurality ofcameras 1522, a microphone 1520, a gesture sensor 1518, a pose sensor(e.g., inertial measurement unit (IMU)) 1516, the display device 1512,and a wireless/wired communication interface 1524. The cameras 1522 ofthe XR headset 150 may be visible light capturing cameras, near infraredcapturing cameras, or a combination of both.

The cameras 1522 may be configured to operate as the gesture sensor 1518by tracking for identification user hand gestures performed within thefield of view of the camera(s) 1522. Alternatively the gesture sensor1518 may be a proximity sensor and/or a touch sensor that senses handgestures performed proximately to the gesture sensor 1518 and/or sensesphysical contact, e.g. tapping on the sensor 1518 or an enclosure. Thepose sensor 1516, e.g., IMU, may include a multi-axis accelerometer, atilt sensor, and/or another sensor that can sense rotation and/oracceleration of the XR headset 150 along one or more defined coordinateaxes. Some or all of these electrical components may be contained in ahead-worn component enclosure or may be contained in another enclosureconfigured to be worn elsewhere, such as on the hip or shoulder.

As explained above, a surgical system includes a camera tracking system1500 which may be part of a computer platform 1500 that can also providefunctionality of a navigation controller 1502 and/or of the XR headsetcontroller 1510. The surgical system may include the imaging devicesand/or a surgical robot 102. The navigation controller 1502 can beconfigured to provide visual navigation guidance to an operator formoving and positioning a surgical tool relative to patient anatomicalstructure based on a surgical plan, e.g., from a surgical planningfunction, defining where a surgical procedure is to be performed usingthe surgical tool on the anatomical structure and based on a pose of theanatomical structure determined by the camera tracking system 1500. Thenavigation controller 1502 may be further configured to generatesteering information based on a target pose for a surgical tool, a poseof the anatomical structure, and a pose of the surgical tool and/or anend effector of the surgical robot 102, where the steering informationindicates where the surgical tool and/or the end effector of thesurgical robot 102 should be moved to perform the surgical plan.

The electrical components of the XR headset 150 can be operativelyconnected to the electrical components of the computer platform 1500through a wired/wireless interface 1524. The electrical components ofthe XR headset 150 may be operatively connected, e.g., through thecomputer platform 1500 or directly connected, to various imagingdevices, e.g., the C-arm imaging device 1304, the I/O-arm imaging device1306, the patient image database 1530, and/or to other medical equipmentthrough the wired/wireless interface 1524.

The surgical system further includes at least one XR headset controller1510 (also referred to as “XR headset controller” for brevity) that mayreside in the XR headset 150, the computer platform 1500, and/or inanother system component connected via wired cables and/or wirelesscommunication links. Various functionality is provided by softwareexecuted by the XR headset controller 1510. The XR headset controller1510 is configured to receive information from the computer trackingsystem 1500 and the navigation controller 1502, and to generate an XRimage based on the information for display on the display device 1512.

The XR headset controller 1510 can be configured to operationallyprocess signaling from the cameras 1522, the microphone 1520, and/or thepose sensor 1516, and is connected to display XR images on the displaydevice 1512 for user viewing. Thus, the XR headset controller 1510illustrated as a circuit block within the XR headset 150 is to beunderstood as being operationally connected to other illustratedcomponents of the XR headset 150 but not necessarily residing within acommon housing or being otherwise transportable by the user. Forexample, the XR headset controller 1510 may reside within the computerplatform 1500 which, in turn, may reside within a housing of thesurgical robot 102, the tracking cameras 200, etc.

Further Definitions and Embodiments

In the above-description of various embodiments of present inventiveconcepts, it is to be understood that the terminology used herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of present inventive concepts. Unless otherwisedefined, all terms (including technical and scientific terms) usedherein have the same meaning as commonly understood by one of ordinaryskill in the art to which present inventive concepts belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of this specification andthe relevant art and will not be interpreted in an idealized or overlyformal sense expressly so defined herein.

When an element is referred to as being “connected”, “coupled”,“responsive”, or variants thereof to another element, it can be directlyconnected, coupled, or responsive to the other element or interveningelements may be present. In contrast, when an element is referred to asbeing “directly connected”, “directly coupled”, “directly responsive”,or variants thereof to another element, there are no interveningelements present. Like numbers refer to like elements throughout.Furthermore, “coupled”, “connected”, “responsive”, or variants thereofas used herein may include wirelessly coupled, connected, or responsive.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. Well-known functions or constructions may not be described indetail for brevity and/or clarity. The term “and/or” includes any andall combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third, etc.may be used herein to describe various elements/operations, theseelements/operations should not be limited by these terms. These termsare only used to distinguish one element/operation from anotherelement/operation. Thus, a first element/operation in some embodimentscould be termed a second element/operation in other embodiments withoutdeparting from the teachings of present inventive concepts. The samereference numerals or the same reference designators denote the same orsimilar elements throughout the specification.

As used herein, the terms “comprise”, “comprising”, “comprises”,“include”, “including”, “includes”, “have”, “has”, “having”, or variantsthereof are open-ended, and include one or more stated features,integers, elements, steps, components or functions but does not precludethe presence or addition of one or more other features, integers,elements, steps, components, functions or groups thereof. Furthermore,as used herein, the common abbreviation “e.g.”, which derives from theLatin phrase “exempli gratia,” may be used to introduce or specify ageneral example or examples of a previously mentioned item, and is notintended to be limiting of such item. The common abbreviation “i.e.”,which derives from the Latin phrase “id est,” may be used to specify aparticular item from a more general recitation.

Example embodiments are described herein with reference to blockdiagrams and/or flowchart illustrations of computer-implemented methods,apparatus (systems and/or devices) and/or computer program products. Itis understood that a block of the block diagrams and/or flowchartillustrations, and combinations of blocks in the block diagrams and/orflowchart illustrations, can be implemented by computer programinstructions that are performed by one or more computer circuits. Thesecomputer program instructions may be provided to a processor circuit ofa general purpose computer circuit, special purpose computer circuit,and/or other programmable data processing circuit to produce a machine,such that the instructions, which execute via the processor of thecomputer and/or other programmable data processing apparatus, transformand control transistors, values stored in memory locations, and otherhardware components within such circuitry to implement thefunctions/acts specified in the block diagrams and/or flowchart block orblocks, and thereby create means (functionality) and/or structure forimplementing the functions/acts specified in the block diagrams and/orflowchart block(s).

These computer program instructions may also be stored in a tangiblecomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instructions whichimplement the functions/acts specified in the block diagrams and/orflowchart block or blocks. Accordingly, embodiments of present inventiveconcepts may be embodied in hardware and/or in software (includingfirmware, resident software, micro-code, etc.) that runs on a processorsuch as a digital signal processor, which may collectively be referredto as “circuitry,” “a module” or variants thereof.

It should also be noted that in some alternate implementations, thefunctions/acts noted in the blocks may occur out of the order noted inthe flowcharts. For example, two blocks shown in succession may in factbe executed substantially concurrently or the blocks may sometimes beexecuted in the reverse order, depending upon the functionality/actsinvolved. Moreover, the functionality of a given block of the flowchartsand/or block diagrams may be separated into multiple blocks and/or thefunctionality of two or more blocks of the flowcharts and/or blockdiagrams may be at least partially integrated. Finally, other blocks maybe added/inserted between the blocks that are illustrated, and/orblocks/operations may be omitted without departing from the scope ofinventive concepts. Moreover, although some of the diagrams includearrows on communication paths to show a primary direction ofcommunication, it is to be understood that communication may occur inthe opposite direction to the depicted arrows.

Many variations and modifications can be made to the embodiments withoutsubstantially departing from the principles of the present inventiveconcepts. All such variations and modifications are intended to beincluded herein within the scope of present inventive concepts.Accordingly, the above disclosed subject matter is to be consideredillustrative, and not restrictive, and the appended examples ofembodiments are intended to cover all such modifications, enhancements,and other embodiments, which fall within the spirit and scope of presentinventive concepts. Thus, to the maximum extent allowed by law, thescope of present inventive concepts are to be determined by the broadestpermissible interpretation of the present disclosure including thefollowing examples of embodiments and their equivalents, and shall notbe restricted or limited by the foregoing detailed description.

What is claimed is:
 1. A camera tracking system comprising at least one processor operative to: receive patient reference tracking information indicating pose of a patient reference array tracked by a patient tracking camera relative to a patient reference frame; determine a local extended reality (XR) headset view pose transform between a local XR headset reference frame of a local XR headset and the patient reference frame using the patient reference tracking information; receive remote reference tracking information indicating pose of a remote reference array tracked by a remote reference tracking camera; determine a remote XR headset view pose transform between a remote XR headset reference frame of a remote XR headset and the remote reference array using the remote reference tracking information; transform a three-dimensional (3D) computer image from a local pose determined using the local XR headset view pose transform to a remote pose determined using the remote XR headset view pose transform which outputs a transformed 3D computer image; and provide the transformed 3D computer image to the remote XR headset for display with the remote pose relative to the remote XR headset reference frame.
 2. The camera tracking system of claim 1, wherein the at least one processor is further operative to transform the 3D computer image from the local pose to the remote pose while the patient tracking camera is remote from the remote reference tracking camera and not positioned to track pose of the remote reference array, and while the remote reference tracking camera is not positioned to track pose of the patient reference array.
 3. The camera tracking system of claim 1, wherein the at least one processor is further operative to: determine a 4 degree-of-freedom (4 DOF) pose of the remote XR headset based on measured movement along three orthogonal axes of the remote XR headset reference frame and rotation about one of the three orthogonal axes aligned with gravitational direction, wherein the operation to transform the 3D computer image from the local pose determined using the local XR headset view pose transform to the remote pose determined using the remote XR headset view pose transform comprises processing the 4 DOF pose of the remote XR headset through the remote XR headset view pose transform.
 4. The camera tracking system of claim 1, wherein the at least one processor is further operative to: obtain remote operator-gesture tracking information from the remote reference tracking camera indicating movement of an object relative to the remote XR headset reference frame by a remote operator wearing the remote XR headset; determine a remote gesture path relative to the remote XR headset reference frame based on processing the remote operator-gesture tracking information through the remote XR headset view pose transform; and transform the remote gesture path to a local gesture path relative to the local XR headset reference frame using the local XR headset view pose transform.
 5. The camera tracking system of claim 4, wherein the at least one processor is further operative to: provide the local gesture path to the local XR headset for display relative to the local XR headset reference frame.
 6. The camera tracking system of claim 4, wherein the at least one processor is further operative to: determine the remote gesture path relative to the remote XR headset reference frame based on tracking movement indicated by the remote operator-gesture tracking information of a hand and/or a stylus which is moved by the remote operator while concurrently viewing the transformed 3D computer image through the remote XR headset relative to the hand and/or stylus being moved.
 7. The camera tracking system of claim 4, wherein the at least one processor is further operative to: select an operational command from among a set of operational commands based on the remote gesture path corresponding to defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths; and provide the operational command to an equipment which is local to the local XR headset.
 8. The camera tracking system of claim 7, wherein the at least one processor is further operative to: select the operational command for relocating an end effector connected to a surgical robot arm that is movable under control of a surgical robot system, from among the set of operational commands based on the remote gesture path corresponding to the defined gesture associated with the operational command for relocating the end effector; determine a present pose of the end effector based on end effector tracking information indicating pose of the end effector tracked by the patient tracking camera relative to the patient reference frame; and control movement of the end effector by the surgical robot system from the present pose to a target pose relative to the patient reference frame based on the operational command for relocating the end effector.
 9. The camera tracking system of claim 8, wherein the at least one processor is further operative to: determine a planned end effector trajectory path from the present pose to the target pose based on at least a segment of the remote gesture path; and control movement of the end effector by the surgical robot system to conform to the planned end effector trajectory path from the present pose to the target pose.
 10. The camera tracking system of claim 8, wherein: the transformed 3D computer image comprises a graphical representation of the end effector displayed based on the remote pose relative to the remote XR headset reference frame and comprises a graphical representation of anatomical structure of the patient displayed based on the remote pose relative to the remote XR headset reference frame; and the at least one processor is further operative to determine a planned end effector trajectory path from a present graphical pose of the graphical representation of the end effector to a target graphical pose of the graphical representation of the end effector based on tracking movement of fingers and/or a hand of the remote operator wearing the remote XR headset relative to the graphical representation of the end effector displayed relative to remote XR headset reference frame, and control movement of the end effector by the surgical robot system to conform to the planned end effector trajectory path from the present pose relative to the patient reference frame to the target pose relative to the patient reference frame.
 11. The camera tracking system of claim 7, wherein the at least one processor is further operative to: select the operational command from among the set of operational commands which control at least one of the following: operational settings of a computer assisted surgical navigation system; operational settings of a surgical robot system; operational settings of medical imaging equipment which is operable to obtain medical images of anatomical structure of the patient; operational settings of intraoperative neuromonitoring equipment which is operable to monitor neural structures of the patient; operational settings of the local XR headset; operational settings of a computer display which is local to the local XR headset; operational settings of a microscope which is local to the local XR headset; operational settings of an exoscope which is local to the local XR headset; operational setting of a lighting apparatus which is operable to illuminate the patient; operational settings of a powered adjustable surgical bed which is operable to support the patient; operational settings of a microscope which is local to the local XR headset; operational settings of anesthesia equipment which is operable to supply anesthesia to the patient; operational settings of a clock and/or timer which is local to the local XR headset; operational settings of communication equipment which is local to the local XR headset; and operational settings of sound equipment which is local to the local XR headset.
 12. The camera tracking system of claim 1, wherein the at least one processor is further operative to: obtain graphical and/or textual information entered by the remote operator and which is displayed by the remote XR headset with a remote information pose relative to the remote XR headset reference frame using the remote XR headset view pose transform; transform the remote information pose of the graphical and/or textual information to a local information pose using the local XR headset view pose transform; and provide the graphical and/or textual information to the local XR headset for display with the local information pose relative to the local XR headset reference frame.
 13. The camera tracking system of claim 1, wherein the at least one processor is further operative to: correlate in time individual ones of video frames of a local video stream received from the patient tracking camera with individual ones of video frames of a remote video stream received from the remote reference tracking camera; control timing when the individual ones of the video frames of the local video stream are provided to the remote XR headset for display based on the correlation; and control timing when the individual ones of the video frames of the remote video stream are provided to the local XR headset for display based on the correlation.
 14. A camera tracking system comprising at least one processor operative to: receive equipment reference tracking information indicating poses of medical equipments and a patient reference array tracked by a tracking camera relative to a reference frame; determine an extended reality (XR) headset view pose transform between an XR headset reference frame of an XR headset and the reference frame using the equipment reference tracking information; obtain operator-gesture tracking information from the tracking camera indicating movement of an object relative to the XR headset reference frame by an operator wearing the XR headset; select an operational command from among a set of operational commands based on the operator-gesture tracking information; and provide instructions to one of the medical equipments based on the operational command that is selected.
 14. The camera tracking system of claim 13, wherein: the tracking camera is part of the XR headset; and the reference frame is the same as the XR headset reference frame.
 15. The camera tracking system of claim 13, wherein the at least one processor is further operative to: determine a gesture path relative to the XR headset reference frame based on processing the operator-gesture tracking information through the XR headset view pose transform, select the operational command from among the set of operational commands based on identifying that the gesture path corresponds to a defined gesture associated with the operational command, wherein the operational commands in the set are associated with different shaped gesture paths.
 16. The camera tracking system of claim 15, wherein the at least one processor is further operative to: select the operational command for relocating an end effector connected to a surgical robot arm that is movable under control of a surgical robot system, from among the set of operational commands based on the gesture path corresponding to the defined gesture associated with the operational command for relocating the end effector; determine a present pose of the end effector based on end effector tracking information indicating pose of the end effector tracked by the tracking camera relative to the reference frame; and control movement of the end effector by the surgical robot system from the present pose to a target pose relative to the reference frame based on the operational command for relocating the end effector.
 17. The camera tracking system of claim 16, wherein the at least one processor is further operative to: determine a planned end effector trajectory path from the present pose to the target pose based on at least a segment of the gesture path; and control movement of the end effector by the surgical robot system to conform to the planned end effector trajectory path from the present pose to the target pose.
 18. The camera tracking system of claim 17, wherein the at least one processor is further operative to: determine the planned end effector trajectory path based on tracking movement of fingers and/or a hand of the operator wearing the XR headset relative to the end effector.
 19. The camera tracking system of claim 15, wherein the at least one processor is further operative to: select the operational command for relocating medical imaging equipment in a room under control of a computer system, from among the set of operational commands based on the gesture path corresponding to the defined gesture associated with the operational command for relocating the medical imaging equipment; determine a present location in the room of the medical imaging equipment relative to the reference frame based on the equipment reference tracking information; determine a target location in the room for the medical imaging equipment relative to the reference frame based on at least a segment of the gesture path; and control movement of the medical imaging equipment by the computer system from the present location to the target location based on the operational command for relocating the medical imaging equipment.
 20. The camera tracking system of claim 14, wherein the at least one processor is further operative to: obtain first graphical and/or textual information from a first one of the medical equipments, and second graphical and/or textual information from a second one of the medical equipments; and display the first graphical and/or textual information through the XR headset with a pose in the XR headset reference frame defined to be adjacent to the first one of the medical equipments and display the second graphical and/or textual information through the XR headset with another pose in the XR headset reference frame defined to be adjacent to the second one of the medical equipments. 